1. Field
The present subject matter relates to a technique for performing correction processing for a distortion generated due to a rolling shutter method on captured image data.
2. Description of the Related Art
Sizes and weights of imaging apparatuses such as video cameras and digital cameras have been reducing more and more, resulting in an increase in opportunities to capture image data by portable imaging apparatuses. In recent years, Complementary Metal Oxide Semiconductor (CMOS) sensors have been used for image sensors of such imaging apparatuses. This is mainly because the CMOS sensors have such advantages that they can easily realize a cost reduction and an increase in the number of pixels, and their power consumption is low.
However, use of the CMOS sensors as the image sensors may lead to generation of a distortion due to a camera shake or a motion of an object (or a shake of an object), in addition to a vibration of captured image data. This is a phenomenon that does not occur when Charge Coupled Devices (CCDs) are used as the image sensors.
Most of the CMOS sensors are configured to read out image data by a rolling shutter method, in which image data is sequentially read out pixel by pixel or line by line (normally, in a horizontal direction, or a vertical direction in some cases). According to this rolling shutter method, each pixel or each line is exposed for a different exposure time period. However, sequentially reading out image data pixel by pixel generates only a negligibly small time difference among exposure starts of pixels in one line, compared to a time difference among exposure starts of lines. Therefore, in the following description, reading out image data pixel by pixel will be treated as if it has a same problem as reading out image data line by line.
When image data is captured by a sensor of the rolling shutter type, if an object moves or a camera shakes during exposure periods from a top line to a bottom line in a screen, the image of the object is deformed due to a time difference in the exposure period of each scanned line, and this appears as a distortion. Hereinafter, this distortion will be referred to as a rolling shutter distortion.
One possible solution therefor is providing a global shutter function, which is a standard function for the CCDs. Another possible solution is providing a plurality of circuits for reading out pixel data so as to use them in parallel or assign each of them to each region in image data, thereby speeding up the readout operation to reduce a time difference among the exposure periods.
However, the former solution requires transistors of several times as many as pixels, thereby leading to a cost increase. Further, this proposal has a problem of negatively affecting an image quality itself because this prevents an increase in aperture ratios of the pixels. Therefore, adoption of this solution is limited to some imaging apparatuses designed for measurement. On the other hand, the latter solution also leads to a cost increase for providing a large number of readout circuits enough to reduce a time difference among the exposure periods, because the number of pixels in image data is ever increasing.
Therefore, a correction relying on image processing predominates as a method for correcting a rolling shutter distortion. Various methods are proposed as rolling shutter distortion correction processing. For example, many of them are configured to realize the rolling shutter distortion correction processing by geometrically deforming image data, as discussed in Japanese Laid-Open Patent Application No. 2006-186481.
On the other hand, the imaging apparatus includes a sensor that detects a camerawork regarding a rotation and a shift of the imaging apparatus such as a gyroscope and an acceleration sensor. In an image analysis, a camerawork (a self-motion), a global motion, and a motion of an object are calculated or estimated using an analysis technique among a plurality of continuous image data pieces. “Removing Rolling Shutter Wobble” written by Simon Baker et al. presented in CVPR 2010 discusses a method for acquiring motion information by fitting processing between polynomial interpolation of a motion and image data.
A simple correction method is a method for performing the rolling shutter distortion correction processing while focusing on only a translation motion of image data in the horizontal direction and the vertical direction. On the other hand, a complicated correction method is a method for taking all types of cameraworks and depth positions of an object into consideration according to an existing principle such as Structure From Motion (SFM). A further complicated correction method is a method for performing the rolling shutter distortion correction processing while discriminating a motion of an object to estimate or calculate motion information. In this manner, the rolling shutter distortion correction processing makes a correction that can reduce a rolling shutter distortion by considering a time difference in an exposure period of each line, calculating a geometric deformation that can cancel out it, and applying this geometric deformation to image data.
Regarding a temporal direction, there are also simple methods directed to only a distortion (generally categorized as a shear) generated by a motion that can be considered to have a constant direction and size as a motion during an exposure period in one frame. On the other hand, there are also methods directed to a distortion (categorized as a wobble or a jello effect) generated by a motion having a direction and a size that are not constant but variable during a frame period.
Further, for actually implementing the rolling shutter distortion correction processing, there are various kinds of possible methods such as a real-time method in which the rolling shutter distortion correction processing is performed at the time of imaging, an off-line method in which the rolling shutter distortion correction processing is performed at the time of reproduction or on recorded image data, and a method in which the rolling shutter distortion correction processing is performed in another image processing apparatus than the imaging apparatus.
Any of the above-described methods perform the rolling shutter distortion correction processing by setting a single line as a reference line among lines having different exposure periods, and geometrically deforming image data to eliminate a rolling shutter distortion caused by a motion generated during a different exposure period as if the lines are imaged during the same exposure period as the reference line, which is in common among all of these methods.
This rolling shutter distortion correction processing requires a determination of a certain row as a reference row, and calculation of how much other rows deviate from this reference row. Known configurations include selecting a first row as the reference row in consideration of real-time processing, and selecting a central row in image data as the reference row to reduce a deformation amount of the whole image data. In either case, the reference row is fixed.
Further, the geometric deformation for canceling out a motion also requires imaging with a larger size than output image data by adding a margin region to the size of the output image data, so as not to generate a region having no pixel data regardless of which correction method is employed.
Regarding this margin region, if an electronic camera shake correction is performed at the same time, a margin region for this electronic camera shake correction should be also taken into consideration. The electronic camera shake correction is a correction of shifting a position where image data is read out according to a degree of a camera shake from the image data captured while a margin region is added thereto. The camera shake correction is realized by changing a position where the image data is read out from a sensor, cutting out output image data from a buffer storing the image data, or geometrically deforming the image data stored in the buffer.
However, according to the above-described conventional rolling shutter distortion correction processing, as a rolling shutter distortion is larger, a larger margin region is necessary to correct the rolling shutter distortion. On the other hand, a request for an increase in the number of pixels in image data from users leads to a tendency to reduce the number of sensor pixels assignable to the margin region for the rolling shutter distortion correction processing. This tendency is further remarkable in the methods that also perform the electronic shake correction at the same time, under the influence of a trend of making a vibration-proof function more dynamic.
Therefore, one object for the rolling shutter distortion correction processing is to realize saving of the margin region required for the rolling shutter distortion correction processing while enabling a correction of even a rolling shutter distortion generated by a larger shake at the same time. Especially, if the rolling shutter distortion correction processing is performed in addition to the electronic vibration correction mechanism, the margin region should be shared therebetween.
Therefore, it is desirable to reduce the margin region required during the rolling shutter distortion correction processing.